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• Why Human Performance Modeling (HPM)? 

• Importance of HPM to System Design 

• HPM Tools and Applications 

- CogTool 

- C3TRACE 
-IMPRINT 

- MIDS Plug-in 

• Expansion of tools 

• Summary 
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Why Human Performance 
Modeling (HPM)? 


Concept System 

Many Variables 

Field Study Not Feasible 

Too Dangerous 

Model - Test - Model 

System Performance s /(human performance) 
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Importance of HPM to System 
Design 


User Needs 


Technology Opportunities & Resources 


LRwter Secretary of Defense far Aflqutidion. Technology, and Logistics, (200S) Department 
o! the Defense Instruction (DoDl) 5000.02 Operation oftte Defei»AoqiriMiM System 
Available online: ti Ittrifoww dfc mi lArti s/ufi recii '/a sfoa mes/pdlfiOO D Q2p . cdf 
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Materiel 

Solution 

Analysis 

Technology 

Development 

Engineering and 
Manufacturing 
Development 

Production & 
Deployment 

Operations & 
Support 
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/ \ Sustainment / 


^Fre-Systems Acqmsition/ ^\ 


Systems Acquisition 


Decision Point /\= Milestone Review ( ; = Decision Point if PDR is not conducted before Milestone B 



Provide quantitative data to inform trade off 
decisions early in design process 
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CogTool 



1 . Set i L to compare design altam atires on a suite of leaks 


General purpose Ul prototyping tool 
Automatically evaluates design with 
predictive human performance model 
"cognitive crash dummy" 



Bonnie E. John, Principle Investigator 
Hu man- Computer Interaction Institute 
School of Computer Science 
Carnegie Mellon University 
http://cogtool.org 


6 Examine what the ACT -R model 0 c to produce the prediction 
in an inlaractivn timeline v sua izalian 


John, B. E, n (2010) CogTool: Predictive Human Performance Modeling by Demonstration, Proceedings of tlie 19th Annual Conference on Behavior 
Representation in Modeling and Simulation (BRIMS) (Charleston, SC, March 21-24, 2010). 
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CogTool Application 
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Compared time to 
complete programming 
tasks within two 
environments 

• 2002 - Unix command 
line and Vim editor 

• 2010 - Eclipse Parallel 
Tools Platform (FTP) 


AnO ' Project : PERtS.tLLPTP _tom parison_2 0 09 1 2 1 6_ 1 5 13 


Results 

Eclipse PTP interface will improve 
performance of skilled programmers 
over 2002 command line interface 


Tasks 

Command Line 

FTP- Ed ipse 


It KeliowWorldjnpi 

Min; 114,405 S 

Min; 40,090 s 


HellOWorldjrtpi with keyboard 

161,111 s 



HelloWofld_mpi with mouse 

114,405 s 

40-090 S 


¥ Fl Help 

Min: 30,726 s 

Min: 10.462 s 


FI Help with keyboard 

30,726 s 



FI Help with mouse 

31.247 S 

10.462 S 


t Code Folding 

Min: 5,780 s 

Min: 3,563 s 


Code folding with keyboard 

10,490 s 



Code folding with mouse 

5,780 s 

3.563 S 


Barrier Analysis 

40 ,149 s 

11.833 s 



Richards, J., Bellamy, R„ John, B-, Swart, C. & Thomas, J, (2010) Using CogTool to Model Programming Tasks. Psychology of 
Programming Interest Group WIP (PPIGWIP) (Dundee 2010), www.ooio.oro . 
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C3TRACE 


Command, Control, and Communications • Techniques for the Reliable 
Assessment of Concept Execution 


Goal: To conduct “what-if’ analyses based on 
information flow and quality, to discover alternative 
organizational, personnel, and system 
configurations that increase performance 

• Evaluate effects of different 



personnel architectures and 
information technology on system 
and human performance 

• Investigate efficiency and 
effectiveness of message 
processing in Command & 

Control environments 
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C3TRACE Application 


Future Command and Control Cell Analysis 


System 

Issue 


Requirements and Force 
Design are in conflict 



1 , Modify deployment concept 

Require different aircraft 

2, Accept degradation in 16 soldier C2 cell 

Reduced capability 

3, Accept more C2Vs 

More money, maintenance. Sifts 



Results 

Performance Measures 

Utilization 

Probability of “good decision" 
Messages handling 


6 Cell Configuration'll 



• 1 9 of 24 - 1 00% utilization • 1 3 of 24 - 1 00% utilization 

• 6 of 24 - 25+% poor decision quality • 5 of 24 - 25+% poor decision quality 

• 18 of 24- dropped 50+% of messages • 8 of 24 -dropped 50+% of messages 


Mitchell, D, K., Samms, C, Kozycki, R. t Kilduff, P. p Swofooda, J„ BAflimashaun, A. (2006) Soldier Menial Workload, Space Claims t and Information Flow 
Anatysis of the Combined Arms Battalion Headquarters Command and Control (C2) Cells (ARL-TR-3861). Amy Research Laboratory, APG, MO. 
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Improved Performance 
Research Integration Tool 
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PerforEii*n<# taemdi iMcfntten iw>i 



334 users supporting Army, Navy, Air Force , 
Marines < NASA , Department of Homeland 
Security (DHS). Department of Transportation 
(DoT), Joint and other organizations 
across the country 



http://www.arLarmy.mil/IMPRINT 
https ^/km3.aiionsciencexom/sites/im print 


IMPRINT can b© us©d to 


• Set realistic system 
requirements 

• Identify future manpower & 
personnel constraints 

• Evaluate operator & crew 
workload 

• Test alternate system-crew 
function allocations 

• Assess required 
maintenance man-hours 

• Assess performance during 
extreme conditions 


• Examine performance as a 
function of personnel 
characteristics and training 
frequency & recency 

• Identify areas to focus test and 
evaluation resources 

• Quantify human system 
integration risks in mission 
performance terms to support 
milestone review 

• Represent humans in 
federated simulations 


IMPRINT is a trade-off analysis tool 
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IMPRINT Application 


System 

Issue 



Methodology 

Identified functions and tasks via 

knowledge elicitation 

Set up experimental conditions to 

model based on varying function 

allocations 

Built models 

Validated models by walking-through 
with Soldiers 

Completed runs and prepared 
results 


Results 


Commander - Driver and Gunner 

hbjtvHi ftcriJoad of all conttilions 



Gunner - Driver and Commander 

Mo shooting on Hie n?,-t 




X 

▼ 

Commander - Gunner and Driver 

| ty»e ci wnn*fnt*t fijncton ntoMhtfi, vttfn* comoiirdst 


i 

Commander Driver and Gunner 

Tuna cre*memberc scanning. ato*$ hunei kflw phtasophv 



Mitchell, D. K.. Saimnrts, C. L„ Henthom, T.< £ Wojctechowski, J. 0. (2003), Trade study: A Two- versus three- Soldier crew for the Mounted Combat System 
(MGS) and other future combat System platforms (Technical report ARL-TR-3026). Aberdeen Proving Ground. MD„ U.S. Army Research Laboratory. 



MO DStM WORLD 

Coflfiwcncfl- ft £*po 


A Decade of Impact on 
Soldier-System Integration 


Combined Arms Testbed 

First identification of workload issues 
associated with a 2 Soldier common crew 


Command and Control Cell 

Supported requirement for 24 
personnel allocated to the battalion in 
the Unit Reference Sheet 


Lightweight Howitzer 

Supported the possibility of 
reducing crew size 


Future Howitzer 

Workload issues associated with 
rearming resulted in an 
automated rearming concept to 
be included in system design 


Autonomous 
Navigation System 

Provided support for the ANS 
technology to increase crew 
performance 


1999 


2001 


2003 


2005 


2007 


2009 


Situational Understanding 
STO 

Identified critical information 
requirements for system and 
display development 


Future Tank Platoon 
Leader Variant 

High workload analysis 
predictions matched 
ex pen mental results 


All Future Concept 
Vehicle analyses 

Soldier workload identified as 
#1 issue during preliminary 
design review 


Future Tank 

identified workload issues which 
resulted in system design change 
from 2 to Z Soldier crew 


Future Reconnaissance and 
Surveil lance Vehicle 

Served as basis of manning 
assessment and justified need for all 
operators to have displays 
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iflwwaB" Multimodal Information Design 
Support (MIDS) Tool Plug-in 

* Develop potential mitigation strategies from multimodal 
design guidelines matched to areas of high workload as 
identified in IMPRINT 



Hasssajii"" Expansion of Tools 

• Develop smart “links” between tools 

• Keep up with evolving analysis demands 

• Specific Enhancements 

- CogTooi 

• Additional measures 

- C3TRACE 

• Visualization of impacts to decision quality 

- IMPRINT 

• Connect to system engineering 

- MIDS Plug-in 

• Predict effect of incorporating mitigation strategies 
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Summary 
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• Use of human performance modeling tools can 

- Provide quantitative data to inform trade off decisions 
early in design process 

- Cost savings 

- Better design 

- Focus test and evaluation resources 

- Model - Test - Model approach 

• Expand tools to answer new analytic 
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